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ABSTRACT 



The authors establish the fundamental lemma for a relative trace formula. This 
trace formula compares generic automorphic representations of GSp(4) with auto- 
morphic representations of GL(4) which are distinguished with respect to a char- 
acter of the Shalika subgroup, the subgroup of matrices of 2 x 2 block form ^ 9 X ° ^ . 

The fundamental lemma, giving the equality of two orbital integrals, amounts to a 
comparison of certain exponential sums arising from these two different groups. On 
the GSp(4) side one has the Kloosterman sums for this group, and on the GL(4) 
side certain, new, relative Kloosterman sums. To show that these are equal for 
each relevant Weyl group element, the authors compute the Mellin transforms of 
the sums and match them in all cases. 
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CHAPTER I 



INTRODUCTION AND STATEMENT OF RESULTS 



1. Introduction 



(1.1) Let K be a number field and let G be the group GL(4) regarded as an 
algebraic group over K. Suppose that ir is a cuspidal automorphic representation 
of G. Thus, conjecturally, tc is associated to a representation r of the hypothetical 
Langlands group Lk into L G° = GL(4, C). Suppose furthermore that 7r is self- 
contragredient. Then either the symmetric square L-function or the exterior square 
L- function of 7r has a pole at s = 1. Assume that the exterior square L-function has 
a pole at s = 1. Conjecturally, this means that A 2 r has a fixed vector, and hence 
that r factors through the symplectic group in 4 variables Sp{A, C), which is the L- 
group of 50(5, K). Thus it is predicted that tt is the functorial image of a cuspidal 
automorphic representation n' (or rather a packet of representations) of the group 
SO(5). On the other hand, the condition that the exterior square L-function has 
a pole at s = 1 can be read directly on the space of the representation n, due to 
the integral representation for the exterior square L-function given by Jacquet and 
Shalika [JS]. More precisely, let H be the Shalika subgroup of GL{4) 



again regarded as an algebraic group over K. If ip is an additive character of the 
adeles A of K trivial on K define a character still denoted ip of H (A) by 



Then the exterior square L-function has a pole at s = 1 if and only if 7r is distin- 
guished with respect to ip, in the sense that there is a vector (f> in the representation 
7r such that 



More generally, we can consider automorphic representations which are essen- 
tially self-contragredient, that is, self-contragredient up to twist by a character. 
This amounts to replacing the exterior square L-function in the above discussion 
by a suitable twist. A pole at s = 1 conjecturally implies that the representation r 
factors through the group of symplectic similitudes GSp(4, C), which is the L-group 
of GSp(4 7 K). This means that the representation n should be distinguished with 
respect to a character of the form h h- > x(detgf) ip(trX). 




(1.1) 
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(1.2) The above considerations suggest the existence of a trace formula of the fol- 
lowing type. For simplicity, we restrict ourselves to the study of self-contragredient 
representations with trivial central character, and we consider only the character ip 
of the Shalika subgroup. Let $ be a smooth function of compact support on G(A). 
Define as usual a kernel function 

K^(g u g 2 ) = ^(91^92). 

Similarly, let G' be the group GSp(4) regarded as an algebraic group over K, let $' 
be a smooth function of compact support on G'(A), and define the kernel function 

K*,(g' 1 ,g' 2 ) = £ ^Vi" 1 ^)- 
The trace formula in question takes the form: 

(1.2) J K< i> (h~ 1 ,n)ifj(h)9(n)dhdn = J K^( t n 1 ,n 2 z) 9'(n^ 1 n 2 ) dn 1 dn 2 dz. 

Here h is integrated over the Shalika subgroup, n, ni,n 2 are integrated over the 
unipotent radicals of the respective Borel subgroups, 9, 9' are nondegenerate char- 
acters, and z is integrated over the center of GSp(4). Both sides can be computed 
in terms of orbital integrals, that is, integrals taken over double cosets of the ap- 
propriate groups. Not all double cosets contribute. There is a bijection between 
the relevant double cosets of the two groups, and the above identity holds if the 
corresponding orbitals integrals are equal. This is described in more detail below. 
Assuming that there are enough pairs $, $' satisfying this condition of matching 
orbital integrals, spectral expansions of the kernels should allow one to derive from 
this identity another identity asserting that 

J K cusp (h~\n)ij(h)9(n) dhdn= J K cusp (V, n 2 z) 9' {n^n 2 ) dmdn 2 dz. 

Here the exponent cusp is used to indicate that only the cuspidal part of the kernel 
is considered. (In fact, one expects that some discrete terms will have to be added 
to the cuspidal kernel.) 

What are the applications of such an identity? Via the twisted trace formula, 
one expects to be able to establish that (essentially) self-contragredient automorphic 
representations of GL(4) whose (possibly twisted) exterior square L- function has 
a pole are functorial images from the group GSp(4). The present trace formula 
would then establish that in any global packet of representations of GSp(4) there 
is at least one element with a global Whittaker model. 

We caution that the details of this program are far from completed. In particular, 
we have not discussed the relation between representations of SO (5) and GL(4). 
In the case at hand, the Weil representation might be a quicker way to arrive at 
the same result. However, for higher ranks, the Weil representation is not available 
and one has to resort to a trace formula of the above type. In that case the group 
GSp{4) is to be replaced by a spin group whose L-group is the GSp(2n, C). 
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(1.3) If $ and $' are each products of local functions $„, the matching 
of orbital integrals for (1.2) reduces to a local question. At almost all places the 
functions <&' v are the characteristic functions of the respective maximal compact 
subgroups. In this paper, we match the orbital integrals of these characteristic 
functions, that is, we establish the "fundamental lemma" for the trace formula at 
hand. The orbital integral on GSp{A) is of Kloosterman type, while the orbital 
integral on GL(4) is a new sort of relative Kloosterman integral. This matching 
is the first crucial step in establishing the formula (1.2). However, our method of 
proof is rather computational, and thus leaves something to be desired. Essentially, 
we have to show that functions of two (or one) variables are equal. We do so 
by showing that their formal Mellin transforms are equal, and this is a series of 
lengthy calculations. It is remarkable that the resulting formulae are relatively 
simple. Thus, it is possible that the Mellin transforms of our functions have some 
independent interest. 

This use of formal Mellin transforms was introduced, in the context of quadratic 
base change on GL(2), by Jacquet and Ye [JY1]. A number of additional papers 
related to the relative trace formula are listed in the References. 

The main theorem of this work is given below in Section 4 of this Chapter. This 
theorem was announced in [FJ]. 

(1.4) In the rest of this paper we pass to the local situation. Throughout we fix 
the following notation. Let K now denote a nonarchimedean local field of residual 
characteristic not equal to 2, O be its ring of integers, and w be a local uniformizer 
of K. Let q denote the cardinality of the residue field O/wO, and | | denote the 
normalized absolute value on K, so that \w\ = q~ l . Fix an additive character ip 
of K with the property that the largest fractional ideal on which ip is trivial is 
precisely O. 

(1.5) Equations are numbered consecutively throughout each section. The fifth 
numbered equation of Chapter 1, Section 2 is referred to as (2.5) throughout Chap- 
ter 1, and as (1.2.5) in the other chapters. The numbering of Lemmas, Propositions, 
and Theorems (together) is similar. Sections are also divided into numbered para- 
graphs, for the convenience of the reader. 

2. The Shalika Subgroup H of GL(4): Relevant 
Double Cosets and Kloosterman Integrals 

(2.1) For convenience, let us write H instead of H(K) for the Shalika subgroup 
of GL(4,K): 

g eGL(2,K),XeM 2x2 (K)Y 

If B is a Borel subgroup of GL(2,K), let H(B) be the subgroup of the Shalika 
subgroup defined by 

beB,X eM 2x2 (K)\. 
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When working with the Shalika subgroup (this Section and Chapters 2 and 3), we 
let N denote the subgroup of GL(4, K) consisting of upper triangular unipotent 
matrices. 

The additive character ip gives rise to a character of the group if, which we shall 
again denote by ip, by the formula (1.1). In addition, let 6 denote the character of 
iV given by 

9{n) = ip{n 1)2 + n 2 ,3 + n 3j4 ). 

(2.2) A double coset HrN of 67L(4, K) is said to be H-relevant if the map 

hrn I— > ip(h)6(n) 

is well defined. This is the case if and only if 

(2.1) 9{n) = ipirnr' 1 ) for all n G r~ l Hr n N. 

It is precisely these double cosets which enter into the relative trace formula. 

To make these explicit, let W denote the Weyl group of GL(4) with respect to 
the standard Borel subgroup B 4 . Then W is isomorphic to the permutation group 
54. To make this explicit, associate with each permutation a G S4 the permutation 
matrix with entries 1 in the (i,a(i)) positions and elsewhere. Note that with 
this convention, permutations are multiplied from the right, so (12)(13)=(123) 
rather than (132), and if w is the permutation matrix corresponding to a, then 
w~ 1 mijw = m^jj^jj). Let diag( ) denote the 4x4 diagonal matrix. Then we have 

Proposition 2.1. The H-relevant double cosets ofGL(4,K) are 

(1) #(1243)iV 

(2) #diag(-ai,ai,l,l) N 

(3) H (132) diag(ai, 1, 1, 1) N 

(4) #(12)diag(aia 2 ,ai,l,l)AT 

with ai, a 2 G K x . 

Proof. We first determine a complete set of coset representatives for the double 
cosets H\GL{4, K)/B 4 . Let P 2 ,2 be the standard parabolic subgroup of type (2,2), 
and let W 2 ,2 be the subgroup of W corresponding to P 2)2 . Identify W with the 
group of 4 x 4 permutation matrices as above. By the Bruhat decomposition, we 
have the disjoint union 

GL(4,K)= |J P 2 ,2wB 4 . 

wew 2 , 2 \w 

Now a set of coset representatives for i7\P 2 2 is given by matrices of the form 

^ j ^ with g G GL(2,K). Such a g may be written g = &i«;i& 2 with w\ a 

2x2 permutation matrix and b± and 6 2 either upper or lower triangular (with 
independent choices). Changing on the left by an element of the Shalika subgroup, 
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But for correct choices of upper/lower triangularity of the bi, they may each be 
pulled past the w and absorbed in B 4 . It follows that for any w G W , 

P 2 ,2wB 4 = HwB 4 U H(12)wB 4 . 

This gives the decomposition 

(2.2) GL(4,K) = (J HwB 4 . (disjoint union) 

<(34)>\VK 

Next let us determine which of the twelve double cosets HwB 4 in (2.2) may 
contain a relevant double coset HwaN ', for some diagonal a. Suppose that w 
corresponds to a permutation a. Consideration of the characters tj; and 9 shows 
that (2.1) holds only when the following conditions are satisfied: 

(1) If <x(3) < <x(l), then a(l) = a(3) + 1. 

(2) If a(4) < (7(2), then a{2) = a(4) + 1. 

(3) If a(3) < a(2), then a{2) ^ a(3) + 1. 

(4) If a(4) < a(l), then a(l) ^ a{4) + 1. 

(5) If a(l) < a(2) and <r(3) < <r(4), then either a(l) + 1 < a(2) and a(3) + 1 < 
a(4), or a(l) + 1 = (7(2) and a(3) + 1 = (7(4). 

Using these conditions, one finds that if a double coset HwaN is relevant, then 
(modulo left multiplication by (34)), a must be one of the permutations 1, (12), 
(132), and (1243). 

To complete the proof of the Proposition, one checks that for each of these four 
permutations, the double cosets of the form HwaN satisfying (2.1) are precisely 
those of the form listed. We omit this calculation. 

(2.3) For each relevant double coset, we now introduce an integral of the unit 
element S of the GL{4) Hecke algebra. The integral is to be compared with an ap- 
propriate symplectic integral. To facilitate this comparison, it is useful to consider 
a Mellin transform of the integral against a quasicharacter of the appropriate torus. 

The integral is described as follows. Let S be the characteristic function of the 
maximal compact subgroup /C = GL(4,0) of GL(4,K). Suppose that the double 
coset HwaN is relevant. Then we consider 

(2.3) F(w,a)= / / Z(hwan)i;(h)9(n)dhdn, 

where dh is the quotient measure obtained from the normalized right Haar measure 
on H. 

A first simplification of (2.3) is given as follows. Let B denote a Borel subgroup of 
GL(2, K), and let ICi denote the compact subgroup GL{2, O). Then with suitable 
normalizations of the Haar measures 

/ dg = dki / db. 

JgL(2,K) JK x JB 
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k\ 

Moving the matrix I . 1 e JC to the left and using the /C-invariance of H, we 



fci 

may do the /Ci integral. We find that the integral (2.3) is unchanged if we replace 
H by the subgroup H(B). 

We shall compute the integral F((1243), I4) directly momentarily. To study 
the remaining integrals, let X i,X2 be quasicharacters of K. Let us introduce the 
following formal Mellin transforms of F(w,a): 

(2.4) 

M F ( X i; 1) = / F (1, diag(-ai, a u 1, 1)) X i(ai) ^ x «i 

(2.5) 

M F ( X i; (132)) = / F ((132), diag(a l5 1, 1, 1)) X iM d x ai 
(2.6) 

M F ( X i, X2; (12)) = / F ((12), diag(aia 2 , a h 1, 1)) X i(ai) X 2 (a 2 ) d x ai rf x a 2 . 

J(RX) 2 

The meaning of these formal integrals is the following. Let 4>a be the characteristic 
function of the compact set q~ A < \x\ < q A , and introduce the integrals 

(2.7) M F (xi;l;A)= [ F (l,diag(-ai, a u 1, 1)) X iM 4aM d><a u 
(2.8) 

M F (xi; (132); A) = f F ((132), diag(a l7 1, 1, 1)) X i(ai) ^(ai) d x a u 
Jrx 

M f (xi,x* (12); A) = / F ((12), diag(a!a 2 , a 1; 1, 1)) X i(ai) X2 (a 2 ) 
(2.9) x A (ai) </>^(a 2 ) d x ai rf x a 2 . 

The integrands then have compact support. If the quasicharacters X i, X2 are ram- 
ified, then these integrals have limits as A tends to infinity. The formal Mellin 
transform is by definition this limit. In (2.7) and (2.8), when the quasicharacter 
Xi is unramified, these integrals represent Laurent polynomials in X i(ro). As A 
tends to infinity, these Laurent polynomials approach a Laurent series; this is by 
definition the formal Mellin transform. Here the topology on the space of Laurent 
series is that of pointwise convergence of the coefficients. The treatment of (2.9) is 
similar; the formal Mellin transform is a Laurent series in Xi( w ) an d Xzipv) when 
both quasicharacters are unramified, and is a Laurent series in one variable in the 
mixed cases. 

For a fixed relevant w, we occasionally suppress the dependence of M F on w. 

(2.4) The quantity M F {xu Xi'-, (12)) is evaluated in Chapter 2 of these notes. In 
Chapter 3 we evaluate the remaining two Mellin transforms. First, though, let us 
dispose of F((1243),/ 4 )- 
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Proposition 2.2. F((1243),/ 4 ) = 1. 

Proof. A direct computation shows that with w = (1243) 



HnwNw' 1 



/l x 
1 

V 



1 X 

1/ 



r s 1 
u 1 



Replacing H by H(B) where B is the standard Borel subgroup of GL(2) consisting 
of upper triangular matrices, we thus arrive at the expression 



z/ 1 



F((1243),/ 4 ) 



-7 



x 0(n) 




\ 



1/ 

h 
U 



n 



u 



\ 

(1243)n 



J 



d'jd x t 3 d x t4 : dn 



9{n) 



tj J 



d'yd x t 3 d x t 4: dn, 



where for the last step we have moved (1243) to the left and used the /C-invariance 
of S. But a matrix calculation (or the use of Pliicker coordinates, as in Chapter 2 
below) shows that the matrix 



(U 



V 



lh t 4 



\ 



tj 



is in /CiV if and only if it is in /C. Thus the integrand is nonzero only when n e /C, 
and it is easy to see that the integral reduces to 1. 

3. The Symplectic Group GSp(A): Relevant 
Double Cosets and Kloosterman Integrals 

(3.1) We consider the similitude symplectic group G'(K) = GSp(4,K) for the 
skew symmetric matrix 



(3.1) 

where we have set: 



w 



w, = 



w s 
-w s 



1 

1 



We denote by T the group of diagonal matrices in G'{K). It consists of all matrices 
of the form 



(3.2) 



t = diag(ai, a2, a 2 X A, a 1 X A) . 
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The scalar A is the similitude ratio. If A = 1 the matrix t is in the symplectic group 
and the positive roots take the following values on t: 





X 





\ 





1 














1 


—x 


Vo 








1 / 



f 1 





V 


rX 





1 


s 


V 








1 





Vo 








1/ 



(3.3) 

We let Z be the center of G'(K). When working with the group GSp(4, K) (this 
Section and Chapters 4 and 5), we let A" denote the group of upper triangular 
symplectic matrices with unit diagonal. Thus A" is the semi-direct product of the 
group V of matrices of the form: 



(3.4) v 
and the group U of matrices of the form: 

(3.5) u = 



We define a generic character 6 of iV by 

(3.6) 6{u) = ip(s) , 9(v) = ip(x) . 
Also, let be the opposite (transposed) subgroup to N. 

(3.2) A double coset A^rA^ is said to be relevant if the map 

t n\rn2 i— > ^(^1^2) 
is well defined. This is the case if and only if 

(3.7) 9{n) = edrnr- 1 )) for all n G r^Nr n N. 

It is precisely these cosets which enter into the relative trace formula. Relevant 
double cosets are easily described: as in any split group, they are associated with 
the standard parabolic subgroups P = MU of G' . Let wm be the longest element 
in W H M and t an element of the center of M . Then the double coset of w^f is 
relevant and all relevant double cosets have a unique representative of this form for 
a suitable M. 

More precisely, let w\ and 102 be the following matrices: 
/l 0\ /0 

-1 1 [ 1 

01 ' W2 ~ ooo 
Vo 1/ Vo 1 



(3. 



w 1 



1 0\ 


1 

0/ 



Then the a set of representatives for the relevant double cosets consists of the 
matrices of the form 



(1) t 

(2) widiag(ai,a 2 ,-a 2 ,-a i 

(3) W2 diag(ai, ai, a^~ 1 A, a^~ 1 A) 

(4) ty diag(ai,ai,ai,ai). 



l al) 
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(3.3) Let S' be the characteristic function of GL(4,0) D GSp(4,F). For each 
relevant double coset NwtN we consider the integrals 

(3.9) J(w,t) = / / z! (^n2wtni)6{nin2) dn<i dni 7 

J Nnw-i-Nw\N JN 



(3.10) J'(w,t)= II / E' ( t n2wztni)6(nin2) dn2 dn\dz. 

J Z J Nr\w~ 1 ~Nw\N J N 

Our main result compares these integrals to the integrals F(w, a) introduced in 
paragraph (2.3). 

(3.4) First, for completeness, we record the value of J{wq 1 I 4 ). The proof is 
similar to that given in paragraph (2.4). 

Proposition 3.1. 

J(w ,I 4 ) = 1. 

To study the remaining integrals, let Xi>X2 be quasicharacters of K x . Let us 
introduce the following formal Mellin transforms of the functions J'{w,t): 

(3.11) 

M J (xi;w 1 )= / J' (wi,diag(a, 1,-1, -a" 1 )) xi(a)rf x a, 
Jrx 

(3.12) 

Mj(xi;w 2 )= / J'(w 2 ,diag(a, a, 1,1)) xi(a)(i x a, 
Jrx 

(3.13) 

Mj(xi,X2] 1) = / J' (1, diag(aia 2 , a 2 , 1, a] -1 )) Xifai) % 2 (a2) ^ x ai d x a 2 . 

The quantity Mj(xi,x 2 ; 1) is evaluated in Chapter 4 and the other quantities in 
Chapter 5. 

4. The Main Theorem 

(4.1) To state the main theorem, we let aa <-> wt be the bijection between 
relevant double coset representatives on the two groups described as follows. The 
double cosets wt for GSp(4) are taken modulo the center. Then the bijection is 
given by 

(12) diag(aia 2 , ai, 1, 1) <-> diag(— aia 2 , — a 2 , 1, a^ 1 ) 
diag(— ai, ai, 1, 1) <-> toi diag(ai, 1, —1, — a^ -1 ) 
(132) diag(a 2 , 1, 1, 1) <-> w 2 diag(a 2 , a 2 , 1, 1) 
(1243) <-> w . 

The main theorem states that the orbital integrals introduced above for the two 
different groups are in fact equal. 
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Main Theorem. For the bijection of relevant cosets aa <-> wt, the relative 
orbital integral on GL(4) with respect to the Shalika subgroup is equal to the Kloost- 
erman orbital integral on GSp(4): 



To establish the Main Theorem, we shall compare these two functions of the 
form f (0,1,0,2) (resp. f(a±), /(a^)), by comparing their formal Mellin transforms 
in the sense described above. As we shall show in this work, these formal Mellin 
transforms are in fact equal. For the main relevant orbits, these Mellin transforms 
are computed in Chapters II (Mp) and IV (Mj), and the comparison is also included 
in Chapter IV. For the other two nontrivial relevant orbits, the Mellin transforms 
are computed in Chapters III (Mp) and V (Mj), and the comparisons are included 
in Chapter V. 



(5.1) We end this Chapter with a brief review of Kloosterman integrals and of 
Gauss sums. For r, s G K x , the (GL(2)) Kloosterman integral Kl(r,s) is defined 
by the formula 



Here dx denotes the Haar measure on K, normalized so that O has measure 1. The 
following proposition allows one to evaluate this integral explicitly when \rs\ 7^ 1. 

Proposition 5.1. The Kloosterman integral satisfies the following properties: 

(1) Kl(r,s) = \rs\- 1 Kl(s-\r- 1 ). 

(2) Suppose \rs\ < 1. 

(i) If\r\>l,thenKl(r,s) = (l-q- 1 )\r\- 1 . 

(ii) If \r\ < 1, \s\ > 1, then Kl(r, s) = 0. 
(Hi) If |r| < \w\, \s\ < 1, then Kl(r, s) = 0. 
(iv) If \r\ = \s\ < 1, then Kl(r, s) = —1. 

(5.2) If x is a ramified quasicharacter of K and / is a positive integer, we shall 
say that x has conductor / if x is trivial on the subgroup 1 + w^O of (9 X , but not 
on the subgroup (1 + w^~ x O) fl O x . If x is a ramified quasicharacter of conductor 
/, let G(x) denote the Gauss sum 



F(a,a) = J'(w,t). 



5. Kloosterman Integrals and Gauss Sums 





Then one has 



(5.1) 




where dhj denotes the Kronecker delta, for if h 7^ / then the oscillations give zero. 
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